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Background / Context: 

“Double-dose” eoursework has beeome inereasingly popular as a strategy to improve 
aeademie outeomes of low-performing students. Aeeording to Couneil of Great City Sehools 
(2009), nearly half of large urban distriets today use doubled math instruetion as the most 
eommon form of support for students with low aeademie skills. In ninth grade, it is partieularly 
important to provide effeetive algebra intervention beeause algebra typieally has the highest 
failure rates of all ninth-grade eourses. In addition, as many states and distriets now require 
algebra for all students in ninth grade or earlier, eliminating remedial eoursework altogether, the 
need to assist struggling students has also inereased. 

Chieago Publie Sehools initiated a “double-dose” algebra poliey in 2003, requiring 
students with below average skills to enroll in two periods of algebra — support algebra and 
regular algebra. The double-dose algebra poliey was built on an earlier poliey of requiring all 
ninth-grade students to enroll in algebra. While the “algebra-for-all” poliey immersed many low- 
aehieving students in algebra for the first time, it was less sueeessful in raising test seores than 
antieipated, plausibly beeause many students laeked suffieient mathematieal baekground to 
benefit from algebra instruetion (Allensworth, Nomi, Montgomery, & Lee, 2010). In response, 
the double-dose algebra poliey was introdueed as a way to help low-aehieving students eateh up 
on skills needed for algebra eoursework, while engaging all students in rigorous eoursework. 

An earlier study by Nomi and Allensworth (2009) showed that, using a regression 
diseontinuity design, both the intent-to-treat e and the eompliers average treatment effeets were 
modest but highly statistieally signifieant for students with below-average ineoming math skills. 
The eurrent study extends this earlier work to understand heterogeneity in the policy effects 
across schools as well as mediating mechanisms of the policy that may be related to treatment 
effect heterogeneity. We explore conditions under which schools produced the largest effects. 

Purpose / Objective / Research Question / Focus of Study: 

To understand heterogeneity in the effects of double-dose algebra across schools, we 
focus on two policy implementation features and their variation across schools. The first source 
of variation is the degree to which schools followed the cutoff-based course assignment policy. 
The second source of variation is the degree to which schools segregated algebra classes on the 
basis of students’ prior skills in response to the policy. We ask the following research questions: 
1) to what extent did double-dose algebra effects vary across schools?, and 2) how were these 
variations related to the ways in which schools implemented double-dose Algebra? 

Setting: 

Chicago has the third-largest school system in the United States. The student population 
is about 50% African-American, 38% Latino, 9% White, and 3% Asian. Approximately 85% of 
students are eligible for free/reduced priced lunches. 

Population / Participants / Subjects: 

Our sample consists of first-time ninth graders who entered Chicago high school in 2003. 
We excluded students with identified disabilities because their control condition (i.e., regular 
algebra) differed from regular algebra taken by students without disabilities. The resulting 
sample size is 12,916 students in 60 high schools. Table 1 shows descriptive statistics. 
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Among students without disabilities, 40 pereent of students seored below the 50*'^ 
pereentile on the 8*’^-grade ITBS in mathematies, and the average seore was 55.49. Overall 35 
pereent of students took double-dose algebra in 2003. Complianee with the poliey was quite 
strong but not perfeet: Among below-norm students, nearly 80 pereent of students took double- 
dose algebra, while only 0.41 pereent of above-norm students took the eourse. Student 
aehievement seores eome from the algebra subset of the math portion of the PLAN examination, 
a standardized test developed by Ameriean College Testing (ACT Ine.). The mean of algebra 
seores is 6.53 with the standard deviation of 2.35. 

Intervention / Program / Practice: 

The double-dose poliey required first-time ninth graders with 8**'-grade ITBS math seores 
below the national median to enroll in support algebra in addition to regular algebra. The distriet 
offered two eurrieulum options to double-dose algebra teaehers with eurrieular resourees and 
lesson plans, as well as professional development around how to use the two periods for 
instruetion. Also, to ensure eonsisteney between the two algebra elasses, the distriet made the 
following reeommendations: sehools offer algebra and support algebra sequentially in a day with 
the same teaeher, and students needing support eoursework take the same algebra elasses 
together. To follow the distriet’s reeommendations, sehools tended to ereate separate algebra 
elasses — double-dose algebra for below-average students and regular algebra for above-average 
students. As a result, the double-dose algebra poliey intensified sorting by students’ skill levels. 

To summarize, the poliey brought about simultaneous ehanges in eourse taking and 
elassroom peer ability. Figure 1 shows that the distriet wide double-dose algebra enrollment was 
largely defined by the eligibility eutoff seore. Figure 2 shows that elassroom peer ability was 
substantially higher for students above the eut point than students below the eut point (right 
panel in Figure 2), while there was no sueh diseontinuity during pre-poliey years (left panel in 
Figure 2). Thus, the intervention eonsists of both double-dose algebra instruetion and elassroom 
peer ability, and we attempt to deeompose these two effeets. (Please insert Figure 1 and 2 here). 

Research Design: Regression Diseontinuity Design 

Data Collection and Analysis: 

The first analysis estimates the eity-wide average intent-to-treat (ITT) effeets, as well as 
between-sehool varianee of these effeets and sehool speeifie ITT effeets. The seeond analysis 
estimates the eompliers’ average effeets of enrolling in double-dose algebra. Again, we are 
interested in estimating the eity-wide eompliers’ average effeets, the varianee of these effeets, 
and sehool speeifie average eompliers’ effeets. For the eompliers’ analysis, we employ an 
instrumental variable approaeh; using the eutoff seore as an instrument, we identify double-dose 
enrollment effeets among eomplying students. The varianee and sehool-speeifie effeets are 
estimated using random effeets models, proposed by Raudenbush, Reardon, and Nomi 
(201 1). The third analysis deeomposes the effeet of eourse taking and elassroom peer ability, 
using sehool-by-eutoff seore interaetion terms as instruments. 

Analysis 1. ITT effects. We estimate the ITT effeets of double-dose algebra for student i 
in sehool j, using the following model; 

( 1 ) 
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where rjj is a school-speeifie fixed intereept, fy indicates whether students scored below the 
cutoff (T=l if below the cutoff and T=0 otherwise), Xtj is the ITBS percentile scores centered on 
the cut point (i.e., the 50^*^ percentile), (j)ij is the school-specific linear coefficient forX, (j )2 is the 
curvature in the association between X and M (assumed constant across schools), and Sij is an 
idiosyncratic random error.' Here, we are interested in estimating the overall mean effect, E{B) 

= P, the variance in these effects across schools, Var{B)= Tp, as well as the school-specific 
effects, using the empirical Bayes method (Raudenbush & Bryk, 1985). 

Analysis 2. Compilers ’ average effects of enrolling double-dose algebra. We employ an 
instrumental variable strategy, using the cutoff score as an implement to identify compliers’ 
average treatment effects. We use the following two-stage-least-square models, and the first 
stage equation estimates double-dose algebra enrollment as an outcome; 

= dmj + ^ mij ’ (2) 

where M indicates whether students took double-dose algebra (M=l if enrolled in double-dose 
algebra), and the rest of the notations are defined earlier. The overall mean effect of the policy 
on double-dose algebra enrollment (i.e., the city-wide average compliance rate) is E{ff = y. We 
are also interested in the variance in these effects across schools Var(E) = if, and school-specific 
empirical Bayes estimates. 

The predicted value My from Equation (2) captures expected double-dose algebra 
enrollment for student i in school j — exogenous component of double-dose algebra enrollment in 
school j, generated by scoring below the cut point. The second stage equation uses this expected 
value to estimate the outcome Y. Thus, we have; 

( 3 ) 

We focus our interest on the overall mean and variance of Aj as well as school-specific empirical 
Bayes estimates of those effects. Here, Aj is a combined effects of course taking and peer ability. 

Analysis 3. Mediating effects. Schools varied considerably in terms of the extent to 
which they enrolled students in their assigned coursework as well as the degree to which schools 
sorted algebra classes based on students’ prior skills. By taking advantage of this variation, we 
use school-by-cutoff interactions as instruments, generating 60 instruments to identify the effects 
of the two mediators among compliers. Here, we estimate the first-state equation (2) for M and 
the parallel equation using classroom peer ability C as an outcome. We then use both M^- and Cij 
in equation (3). Note that we are not able to estimate school-specific effects of course-taking and 
class composition given one instrument per school. 

Findings / Results: ITT results. The overall average ITT effect is 0.35 (P<0.001), equivalent to a 
standardized effect size of 0.15. The effect varied significantly across schools; the variance of the 
average ITT is 0.16 (p<0.01). Figure 3 shows the distribution of the school-specific ITT effects 



’ Nomi and Allensworth (2009) provide specification checks on the quadratic function; the authors used pre-policy cohorts to 
specify underlying relationships between the assignment variable X and the outcome Y. 

^ Note that the subscript “m” is added on the parameters and error term of Equation (2) to distinguish these from the 
corresponding parameters of Equation (1) 
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Results on compilers average treatment effects. The result of the first-stage equation (2) 
showed that the eity-wide eomplianee rates were high; the average effeets of the eut point on 
double-dose algebra enrollment were 72%. Complianee rates also varied substantially aeross 
sehools; some schools enrolled less than 50% of students in their assigned course, while many 
schools had relatively high compliance rates of 80% or higher. Figure 4 shows the distribution of 
the school-specific compliance rates (please insert Figure 4 here). 

We then estimated the compilers’ average effect of enrolling in double-dose algebra on 
their test scores. The city-wide average compilers’ effect was 0.50 (p<0.001) with the effect size 
of 0.21. The variance of these effects was .39 (p<0.05). In addition, Figure 5 presents the 
relationships between school-specific ITT effects and school-specific compliance rates. The 
figure shows that schools with similar compliance rates have substantial variability in the ITT 
effects, suggesting that the compilers’ average effects also vary among schools with similar 
compliance rates. ^ 

We also compared differences in the ITT effects and compilers’ average treatment 
effects among schools that differed in the degree to which they sorted algebra classes on the 
basis of students’ initial incoming skills. On average, the effect of the cut point on peer ability 
was -0.24, and importantly, the degree of sorting also differed across schools. Table 2 showed 
that both ITT effects and compilers’ average treatment effects were weaker for schools with 
greater sorting than schools with less intense sorting. 

Results on mediating effects of double-dose algebra coursework and peer ability: The 
results of two-stage least squares showed that the effect of double-dose algebra enrollment was 
0.69 (effect size=0.3) and the effect of peer ability was 0.81(effect size=0.4). An additional 
analysis showed that the school-specific effects of the cut point on course enrollment and peer 
ability were not highly correlated (r= -0.27), see Figure 6. These findings together suggest that 
schools are those that implemented double-dose algebra without creating sorted algebra classes. 

Conclusions: 

It is critical to provide effective algebra intervention for students with low math skills, 
particularly in urban districts where many students enter high schools with math skills well 
below grade level. A prior study by Nomi and Allensworth (2009) showed that double-dose 
algebra was successful in raising test scores for students with below-average math skills. 

We extended this earlier work to understand heterogeneity in the treatment effects across 
schools. We showed that the effects of double-dose algebra varied considerably across schools, 
even among schools with similar levels of compliance with the course assignment policy. 

An unintended consequence of this policy was that schools tended to create separate 
algebra classes for below- and above-average students. As a result, the policy intensified skill- 
based sorting. Importantly, the degree of sorting also varied by schools; gaps in classroom peer 
ability between below- and above-average students were larger in some schools than others, and 
these variations seemed to be related to the effects of double-dose algebra. We showed that 
enrolling in double-dose algebra would benefit learning, but having lower-ability classmates 
would hurt learning. Thus, the overall effects of double-dose algebra seemed to be weaker for 
schools with more intense soring than schools that used less intense sorting. 

The next step of our work is to examine what school characteristics may be related to 
treatment heterogeneity and whether the effects may also vary by different student subgroups. 



^ Note that school-specific complies’ average effects can be derived by dividing ITT effects by compliance rates for 
each school. 
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Appendix B. Tables and Figures 

Table 1. Descriptive statistics 



Variables 


Mean 


SD 


Scoring below the cutoff 


0.40 


0.49 


Taking double-dose algebra 


0.35 


0.48 


ITBS math percentile 


55.49 


22.09 


Algebra scores 


6.53 


2.35 



Table 2. The effect of double-dose algebra 


on algebra 


scores by the degree of sorting 




All schools 




Schools with at least 50% 
compliance 


Degree of sorting 


Degree of sorting 




Low 


Average 


High 


Low 


Average 


High 


ITT 


0.49 


0.35 


0.25 


0.43 


0.35 


0.25 


Compilers' effects 


0.69 


0.53 


0.32 


0.63 


0.53 


0.32 


School N 


19 


19 


22 


13 


19 


22 



Figure 1 . Double-dose algebra enrollment rate by math percentile scores 
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Figure 2. Classroom average skill levels by math pereentile scores 

Pre-policy (2001 and 2002) Post-policy (2003 and 2004) 
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ire 3. Distribution of ITT effects. 
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Figure 4. Distribution of the effect of cutoff on double-dose algebra enrollment. 




The effect of cutoff on double-dose algebra enrollment 



Figure 5. The effect of the cutoff on algebra scores (ITT effects, vartical axis) and the effect of 
the cutoff on algebra enrolment (horizontal axis) 
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Figure 6. The effect of cutoff score on double-dose algebra enrollment and classroom peer ability 
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